Skip to main content

CCTV dataset summary

Following are the main locations of CCTV data.

  1. hydrotrek@gqc.com Google Drive, VS_Research/CCTV directory
  2. MSI machine:
    1. 10TB unionsine1 disk (mounted to /media/gqc/unionsine1) → contains a copy of VS_Research directory
    2. 10TB unionsine2 disk (mounted to /media/gqc/unionsine2) → restructured utility data are being created here under cctv directory.
    3. /home/gqc/cctv/ directory → contains the corresponding db files for the restructured data in #2 above.
note

Also check CCTV Project Summary spreadsheet for more information.

DNV​

Outstanding questions to ask Vannary:​

  • Where do the missing 1811 videos get referenced? How do we know that they exist?

DNV_A (dnv_pacp_A_81)​

  • Content: 81 PACP videos
  • Location: /media/gqc/unionsine1/VS_Research/CCTV/DNV
    • Combined with other DNV datasets
    • No restructuring has been performed
    • Open up /media/gqc/unionsine1/VS_Research/CCTV/DNV/Data/feature_engineer_data/inspection_info_DNV_A_version_2.csv for a list of videos.
  • Used for:
    • training CCTV classification models by Vannary
    • training and testing lightGBM models

270 PACP and 35 PACP datasets contain the same videos found in 81 PACP datasets but with different file names.

DNV_B (dnv_pacp_B_406 + dnv_wrc_B_1650)​

  • Content: 406 PACP videos + 1650 WRc videos
  • Location: /media/gqc/unionsine1/VS_Research/CCTV/DNV
    • Combined with other DNV datasets
    • No restructuring has been performed
    • 406 PACP videos → Check /media/gqc/unionsine1/VS_Research/CCTV/DNV/Data/Video_Lists/dnv_pacp_B_406_01_406.csv for a list of videos.
    • 1650 WRc videos → Check /media/gqc/unionsine1/VS_Research/CCTV/DNV/Data/Video_Lists/dnv_wrc_B_1650_*_*.csv files for lists of videos.
  • Used for:
    • 406 PACP videos are used for training CCTV classification models by Vannary
    • 406 PACP videos training and testing lightGBM models
    • WRC 1650 training and testing lightGBM models.

DNV_C (WRc_1066)​

  • Content: 1066 WRc videos
  • Location: /media/gqc/unionsine1/VS_Research/CCTV/DNV
    • Combined with other DNV datasets
    • No restructuring has been performed
    • Open up /media/gqc/unionsine1/VS_Research/CCTV/DNV/Data/Video_Lists/wrc_1066_*_*.csv files for a list of videos.
  • Used for:
    • training and testing lightGBM models

270 PACP Videos (PACP_270)​

Uploaded March 23rd, 2022

  • Content: 270 PACP videos
  • Location: /media/gqc/unionsine1/VS_Research/CCTV/DNV/Data/PACP_270/Data
    • Combined with other DNV datasets
    • No restructuring has been performed
  • Used for:
    • training CCTV classification models by Vannary

270 PACP and 35 PACP datasets contain the same videos found in 81 PACP datasets but with different file names.

35 PACP Videos (dnv_pacp_35)​

Uploaded May 1st, 2022

  • Content: 35 PACP videos
  • Location: /media/gqc/unionsine1/VS_Research/CCTV/DNV/Data
    • Combined with other DNV datasets
    • No restructuring has been performed
    • Open up /media/gqc/unionsine1/VS_Research/CCTV/DNV/Data/Video_Lists/dnv_pacp_35_*_*.csv files for a list of videos.
  • Used for:
    • training CCTV classification models by Vannary

DNV 791 Dataset​

  • The DNV 791 dataset consists of PACP videos.
  • The condition data for these videos is stored in /media/gqc/unionsine1/VS_Research/CCTV/DNV/Data/condition_data_csv/pacp_792_out_5319_videos_info.csv
  • The models related to the 791 videos are stored here /media/gqc/unionsine1/VS_Research/CCTV/models/DNV.

Some of the 270 PACP and 35 PACP datasets contain the same videos found in 81 PACP datasets but with different file names.

1811 Videos not found in CCTV Details​

TODO: Ask Vannary to know if these belong to a single dataset or multiple datasets.

  • Used for: None

SD1​

SD1_700 (prev: 2023_05_26)​

The .mdb files (access database files) are located in the folders where the videos are present. For example, for the video group 01182019, they are present in 2023_05_26/Received_Data/01182019/MDB.

SD1_114 (prev: SD1_B)​

The .mdb files are located in the same folders that the videos are.

SD1_519 (prev: SD1_C)​

  • Content: 519 videos
  • Location: /media/gqc/unionsine2/cctv/sd1/data/SD1_519/received_data
    • Combined with other SD1 datasets
    • No restructuring has been performed
    • Check /media/gqc/unionsine1/VS_Research/CCTV/SD1/Data/Video_Lists/SD1_C_519_*.csv files for lists of videos.
  • Used for:

The .mdb files' location not found.

Summary: hydrotrekai@gmail.com: Has the 1092 videos from 2023_05_26, and 567 videos along with mdb files in SD1 CCTV May 9, 2023. SD1_519 is a subset of these 567 videos.

hydrotrek@gqc.com contains SD1_B, SD1_C videos.

Process for moving data from gdrive, unionsine1 to unionsine2:​

METRO​

METRO_A​

  • Content: 39 videos
  • Location: /media/gqc/unionsine/CCTV/METRO/Data/
    • Combined with other METRO datasets
    • No restructuring has been performed
    • Check /media/gqc/unionsine/CCTV/METRO/Data/Video_Lists/metro_39_*_*.csv files for lists of videos.
  • Used for:
    • training CCTV classification models by Vannary

METRO_B​

  • Content: 530 videos
  • Location: /media/gqc/unionsine1/VS_Research/CCTV/METRO/Data/
    • Combined with other METRO datasets
    • No restructuring has been performed
    • Check /media/gqc/unionsine1/VS_Research/CCTV/METRO/Data/Video_Lists/metro_B_530_*_*.csv files for lists of videos.
  • Used for:
    • training CCTV classification models by Vannary

METRO_B2​

  • Content: 992 videos
  • Location: /media/gqc/unionsine1/VS_Research/CCTV/METRO/Data/
    • Combined with other METRO datasets
    • No restructuring has been performed
    • Check /media/gqc/unionsine1/VS_Research/CCTV/METRO/Data/Video_Lists/metro_B2_992_*_*.csv files for lists of videos.
  • Used for:
    • training CCTV classification models by Vannary

COV​

COV_A​

  • Content: 92 videos
  • Locations: /media/gqc/unionsine1/VS_Research/CCTV/COV/Data, /media/gqc/T7/cctv/COV/Data, /media/gqc/unionsine2/cctv/cov/data/cov_a
    • COV folder present in VS_Research and in T7 is the same. cov folder present in unionsine2 only contains blurred_frames.
    • No restructuring has been performed
    • Check /media/gqc/unionsine1/VS_Research/CCTV/COV/Data/Video_Lists/COV_*_to_*_videos.csv files for lists of videos.
  • Used for:
    • training CCTV classification models by Vannary
    • Attempted to use for CCTV I&I model training, but there are only 40 I&I frames found in the entire dataset.

EBMUD​

Dataset_100923​

  • Content: 3 videos
  • Location: /home/gqc/CCTV/ebmud/Data/Dataset_100923/
    • No restructuring has been performed
    • Check /home/gqc/CCTV/ebmud/Data/Dataset_100923/Video_Lists/ csv files for the list of videos.
  • Used for:
    • training CCTV classification models by Srujana and Sudhir
    • prediction run with CCTV I&I model.

Directory structure of the ebmud data:​

Dataset_100923
├── settings.yml
├── Received_Data
│   ├── group3
│   │   └── 35.mp4
│   ├── group1
│   │   └── 11.mp4
│   └── group2
│       └── 30.mp4
├── Extracted_Frames
│   ├── group1.zip
│   ├── group2.zip
│   └── group3.zip
├── Video_Lists
│   ├── group2.csv
│   ├── group1.csv
│   └── group3.csv
├── Video_DB
│   └── Video_DB.db
└── Logs
    ├── 2023-10-13--13-37-55.log
    ├── 2023-10-13--13-36-48.log
    ├── 2023-10-11--09-30-53.log
    └── 2023-10-11--09-31-51.log
Dataset_100923_old
├── settings.yml
├── Received_Data
│   ├── group1
│   │   └── 11.mp4
│   └── group2
│       └── 30.mp4
├── Extracted_Frames
│   ├── group1.zip
│   └── group2.zip
├── Video_Lists
│   ├── group2.csv
│   └── group1.csv
├── Video_DB
│   └── Video_DB.db
└── Logs
    ├── 2023-10-11--09-30-53.log
    └── 2023-10-11--09-31-51.log
Last updated on by Pavan